Search for: All records

This paper presents approaches to mean-field control, motivated by distributed control of multi-agent systems. Control solutions are based on a convex optimization problem, whose domain is a convex set of probability mass functions (pmfs). The main contributions follow: 1. Kullback-Leibler-Quadratic (KLQ) optimal control is a special case, in which the objective function is composed of a control cost in the form of Kullback-Leibler divergence between a candidate pmf and the nominal, plus a quadratic cost on the sequence of marginals. Theory in this paper extends prior work on deterministic control systems, establishing that the optimal solution is an exponential tilting of the nominal pmf. Transform techniques are introduced to reduce complexity of the KLQ solution, motivated by the need to consider time horizons that are much longer than the inter-sampling times required for reliable control. 2. Infinite-horizon KLQ leads to a state feedback control solution with attractive properties. It can be expressed as either state feedback, in which the state is the sequence of marginal pmfs, or an open loop solution is obtained that is more easily computed. 3. Numerical experiments are surveyed in an application of distributed control of residential loads to provide grid services, similar to utility-scale battery storage. The results show that KLQ optimal control enables the aggregate power consumption of a collection of flexible loads to track a time-varying reference signal, while simultaneously ensuring each individual load satisfies its own quality of service constraints. 

From the summary: The goal of this article is two-fold: survey the emerging theory of QSA (quasi-stochastic approximation) and its implication to design, and explain the intimate connection between QSA and ESC (extremum seeking control). The contributions go in two directions: ESC algorithm design can benefit by applying concepts from QSA theory, and the broader research community with interest in gradient-free optimization can benefit from the control theoretic approach inherent to ESC. 

Foundational and state-of-the-art anomaly-detection methods through power system state estimation are reviewed. Traditional components for bad data detection, such as chi-square testing, residual-based methods, and hypothesis testing, are discussed to explain the motivations for recent anomaly-detection methods given the increasing complexity of power grids, energy management systems, and cyber-threats. In particular, state estimation anomaly detection based on data-driven quickest-change detection and artificial intelligence are discussed, and directions for research are suggested with particular emphasis on considerations of the future smart grid. 

Over the past decade, there has been significant progress on the science of load control for the creation of virtual energy storage. This is an alternative to demand response, and it is termed demand dispatch. Distributed control is used to manage millions of flexible loads to modify the power consumption of the aggregation, which can be ramped up and down, just like discharging and charging a battery. A challenge with distributed control is heterogeneity of the population of loads, which complicates control at the aggregate level. It is shown in this article that additional control at each load in the population can result in a far aggregate model. The local control is designed to flatten resonances and produce approximately all-pass response. Analysis is based on mean-field control for the heterogeneous population; the mean-field model is only justified because of the additional local control introduced in this article. Theory and simulations indicate that the resulting input--output dynamics of the aggregate has a nearly flat input--output response: the behavior of an ideal, multi-GW battery system. 

Demand dispatch is the science of extracting virtual energy storage through the automatic control of deferrable loads to provide balancing or regulation services to the grid, while maintaining consumer-end quality of service.The control of a large collection of heterogeneous loads is in part a resource allocation problem, since different classes of loads are more valuable for different services. The goal of this paper is to unveil the structure of the optimal solution to the resource allocation problem, and investigate short-term market implications. It is found that the marginal cost for each load class evolves in a two-dimensional subspace: spanned by a co-state process and its derivative. The resource allocation problem is recast to construct a dynamic competitive equilibrium model, in which the consumer utility is the negative of the cost of deviation from ideal QoS. It is found that a competitive equilibrium exists with the equilibrium price equal to the negative of an optimal co-state process. Moreover, the equilibrium price is different than what would be obtained based on the standard assumption that the consumer's utility is a function of power consumption. 

This work is a survey of current trends in applications of PMUs. PMUs have the potential to solve major problems in the areas of power system estimation, protection, and stability. A variety of methods are being used for these purposes, including statistical techniques, mathematical transformations, probability, and AI. The results produced by the techniques reviewed in this work are promising, but there is work to be performed in the context of implementation and standardization. As the smart grid initiative continues to advance, the number of intelligent devices monitoring the power grid continues to increase. PMUs are at the center of this initiative, and as a result, each year more PMUs are deployed across the grid. Since their introduction, myriad solutions based on PMU-technology have been suggested. The high sampling rates and synchronized measurements provided by PMUs are expected to drive significant advancements across multiple fields, such as the protection, estimation, and control of the power grid. This work offers a review of contemporary research trends and applications of PMU technology. Most solutions presented in this work were published in the last five years, and techniques showing potential for significant impact are highlighted in greater detail. Being a relatively new technology, there are several issues that must be addressed before PMU-based solutions can be successfully implemented. This survey found that key areas where improvements are needed include the establishment of PMU-observability, data processing algorithms, the handling of heterogeneous sampling rates, and the minimization of the investment in infrastructure for PMU communication. Solutions based on Bayesian estimation, as well as those having a distributed architectures, show great promise. The material presented in this document is tailored to both new researchers entering this field and experienced researchers wishing to become acquainted with emerging trends. 

Sample complexity bounds are a common performance metric in the Reinforcement Learning literature. In the discounted cost, infinite horizon setting, all of the known bounds can be arbitrarily large, as the discount factor approaches unity. These results seem to imply that a very large number of samples is required to achieve an epsilon-optimal policy. The objective of the present work is to introduce a new class of algorithms that have sample complexity uniformly bounded over all discount factors. One may argue that this is impossible, due to a recent min-max lower bound. The explanation is that these prior bounds concern value function approximation and not policy approximation. We show that the asymptotic covariance of the tabular Q-learning algorithm with an optimized step-size sequence is a quadratic function of a factor that goes to infinity, as discount factor approaches 1; an essentially known result. The new relative Q-learning algorithm proposed here is shown to have asymptotic covariance that is uniformly bounded over all discount factors. 

With increase in the frequency of natural disasters such as hurricanes that disrupt the supply from the grid, there is a greater need for resiliency in electric supply. Rooftop solar photovoltaic (PV) panels along with batteries can provide resiliency to a house in a blackout due to a natural disaster. Our previous work showed that intelligence can reduce the size of a PV+battery system for the same level of post-blackout service compared to a conventional system that does not employ intelligent control. The intelligent controller proposed is based on model predictive control (MPC), which has two main challenges. One, it requires simple yet accurate models as it involves real-time optimization. Two, the discrete actuation for residential loads (on/off) makes the underlying optimization problem a mixed-integer program (MIP) which is challenging to solve. An attractive alternative to MPC is reinforcement learning (RL) as the real-time control computation is both model-free and simple. These points of interest accompany certain trade-offs; RL requires computationally expensive offline learning, and its performance is sensitive to various design choices. In this work, we propose an RL-based controller. We compare its performance with the MPC controller proposed in our prior work and a non-intelligent baseline controller. The RL controller is found to provide a resiliency performance — by commanding critical loads and batteries—similar to MPC with a significant reduction in computational effort.